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Foreword 

Gregory F. Treverton 



It is a rare season when the intelligence story in the news concerns intelli- 
gence analysis, not secret operations abroad. The United States is having such 
a season as it debates whether intelligence failed in the run-up to both Septem- 
ber 11 and the second Iraq war, and so Rob Johnston’s wonderful book is per- 
fectly timed to provide the back-story to those headlines. The CIA’s Center 
for the Study of Intelligence is to be commended for having the good sense to 
find Johnston and the courage to support his work, even though his conclu- 
sions are not what many in the world of intelligence analysis would like to 
hear. 

He reaches those conclusions through the careful procedures of an anthro- 
pologist — conducting literally hundreds of interviews and observing and par- 
ticipating in dozens of work groups in intelligence analysis — and so they 
cannot easily be dismissed as mere opinion, still less as the bitter mutterings 
of those who have lost out in the bureaucratic wars. His findings constitute not 
just a strong indictment of the way American intelligence performs analysis, 
but also, and happily, a guide for how to do better. 

Johnston finds no baseline standard analytic method. Instead, the most com- 
mon practice is to conduct limited brainstorming on the basis of previous analy- 
sis, thus producing a bias toward confirming earlier views. The validating of 
data is questionable — for instance, the Directorate of Operation’s (DO) “clean- 
ing” of spy reports doesn’t permit testing of their validity — reinforcing the ten- 
dency to look for data that confirms, not refutes, prevailing hypotheses. The 
process is risk averse, with considerable managerial conservatism. There is 
much more emphasis on avoiding error than on imagining surprises. The ana- 
lytic process is driven by current intelligence, especially the CIA’s crown jewel 
analytic product, the President’s Daily Brief ( PDB), which might be caricatured 
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as “CNN plus secrets.” Johnston doesn’t put it quite that way, but the Intelli- 
gence Community does more reporting than in-depth analysis. 

None of the analytic agencies knows much about the analytic techniques of 
the others. In all, there tends to be much more emphasis on writing and com- 
munication skills than on analytic methods. Training is driven more by the 
druthers of individual analysts than by any strategic view of the agencies and 
what they need. Most training is on-the-job. 

Johnston identifies the needs for analysis of at least three different types of 
consumers — cops, spies, and soldiers. The needs of those consumers produce 
at least three distinct types of intelligence — investigative or operational, stra- 
tegic, and tactical. 

The research suggests the need for serious study of analytic methods across 
all three, guided by professional methodologists. Analysts should have many 
more opportunities to do fieldwork abroad. They should also move much 
more often across the agency “stovepipes” they now inhabit. These move- 
ments would give them a richer sense for how other agencies do analysis. 

Together, the analytic agencies should aim to create “communities of prac- 
tice,” with mentoring, analytic practice groups, and various kinds of on-line 
resources, including forums on methods and problem solving. These commu- 
nities would be linked to a central repository of lessons learned, based on 
after-action post-mortems and more formal reviews of strategic intelligence 
products. These reviews should derive lessons for individuals and for teams 
and should look at roots of errors and failures. Oral and written histories 
would serve as other sources of wherewithal for lessons. These communities 
could also begin to reshape organizations, by rethinking organizational 
designs, developing more formal socialization programs, testing group config- 
urations for effectiveness, and doing the same for management and leadership 
practices. 

The agenda Johnston suggests is a daunting one, but it finds echoes in the 
work of small, innovative groups across the Intelligence Community — groups 
more tolerated than sponsored by agency leaders. With the challenge work- 
force demographics poses for the Community — the “gray-green” age distribu- 
tion, which means that large numbers of new analysts will lack mentors as old 
hands retire — also comes the opportunity to refashion methods and organiza- 
tions for doing intelligence analysis. When the finger-pointing in Washington 
subsides, and the time for serious change arrives, there will be no better place 
to start than with Rob Johnston’s fine book. 
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Introduction 



In August 2001, I accepted a Director of Central Intelligence postdoctoral 
research fellowship with the Center for the Study of Intelligence (CSI) at the 
Central Intelligence Agency. The purpose of the fellowship, which was to 
begin in September and last for two years, was to identify and describe condi- 
tions and variables that negatively affect intelligence analysis. During that 
time, I was to investigate analytic culture, methodology, error, and failure 
within the Intelligence Community using an applied anthropological method- 
ology that would include interviews (thus far, there have been 489), direct and 
participant observation, and focus groups. 

I began work on this project four days after the attack of 1 1 September, and 
its profound effect on the professionals in the Intelligence Community was 
clearly apparent. As a whole, the people I interviewed and observed were 
patriotic without pageantry or fanfare, intelligent, hard working, proud of their 
profession, and angry. They were angry about the attack and that the militant 
Islamic insurgency about which they had been warning policymakers for 
years had murdered close to 3,000 people in the United States itself. There 
was also a sense of guilt that the attack had happened on their watch and that 
they had not been able to stop it. 

Having occurred under the dark shadow of that attack, this study has no 
comparable baseline against which its results could be tested, and it is difficult 
to identify biases that might exist in these data as a result of 1 1 September. In 
some ways, post- 9/11 data may be questionable. For example, angry people 
may have an ax to grind or an agenda to push and may not give the most reli- 
able interviews. Yet, in other ways, post-9/11 data may be more accurate. 
When people become angry enough, they tend to blurt out the truth — or, at 
least, their perception of the truth. The people I encountered were, in my judg- 
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ment, very open and honest; and this, too, may be attributable to 9/11. In any 
case, that event is now part of the culture of the Intelligence Community, and 
that includes whatever consequences or biases resulted from it. 



Background 

The opportunity to do this research presented itself, at least in part, as a 
result of my participation in a multiyear research program on medical error 
and failure for the Defense Advanced Research Projects Agency (DARPA). 1 
The DARPA research focused on team and individual error in minimally inva- 
sive or laparoscopic surgical procedures. This research revealed that individ- 
ual errors were cognitive rather than purely psychomotor or skill-based. For 
example, some surgeons had trouble navigating three-dimensional anatomical 
space using the existing laparoscopic technology, with the result that these 
surgeons would identify anatomical structures incorrectly and perform a surgi- 
cal procedure on the wrong body part. 

Other individual errors were discovered during the DARPA studies, but, for 
the most part, these were spatial navigation and recognition problems for 
which there were technological solutions. Team errors, unlike individual 
errors, proved to be more challenging. The formal and informal hierarchical 
structures of operating rooms did not lend themselves to certain performance 
interventions. Generally, junior surgical staff and support personnel were not 
willing to confront a senior staff member who was committing, or was about 
to commit, an error. 

The culture of the operating room, coupled with the social and career struc- 
ture of the surgical profession, created barriers to certain kinds of communica- 
tion. For a surgical resident to inform a senior surgeon in front of the entire 
operating room staff that he was about to cut the wrong organ could result in 
career “suicide.” Such a confrontation could have been perceived by the 
senior surgeon as a form of mutiny against his authority and expertise and a 
challenge to the social order of the operating room. Although not universal, 
this taboo is much more common than surgeons would care to admit. Unlike 
individual errors, purely technological solutions were of little value in trying 
to solve team errors in a surgical environment. 

The DARPA surgical research was followed up by a multiyear study of 
individual and team performance of astronauts at the National Aeronautics 
and Space Administration’s (NASA) Johnson Space Center. Results of the 
NASA study, also sponsored by DARPA, were similar to the surgical study 



1 Rob Johnston, J. Dexter Fletcher and Sunil Bhoyrul, The Use of Virtual Reality to Measure Sur- 
gical Skill Levels. 
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with regard to team interactions. Although, on the face of it, teams of astro- 
nauts were composed of peers, a social distinction nevertheless existed 
between commander, pilots, and mission specialists. 

As with surgery, there was a disincentive for one team member to confront 
or criticize another, even in the face of an impending error. Eighty percent of 
the current astronauts come from the military, which has very specific rules 
regarding confrontations, dissent, and criticism . 2 In addition to the similarities 
in behavior arising from their common backgrounds, the “criticism” taboo 
was continually reinforced throughout the astronaut’s career. Virtually any 
negative comment on an astronaut’s record was sufficient for him or her to be 
assigned to another crew, “washed out” of an upcoming mission and recycled 
through the training program, or, worse still, released from the space program 
altogether. 

Taboos are social markers that prohibit specific behaviors in order to main- 
tain and propagate an existing social structure. Generally, they are unwritten 
rules not available to outside observers. Insiders, however, almost always per- 
ceive them simply as the way things are done, the natural social order of the 
organization. To confront taboos is to confront the social structure of a culture 
or organization. 

I mention the surgical and astronautical studies for a number of reasons. 
Each serves as background for the study of intelligence analysts. Astronauts 
and surgeons have very high performance standards and low error rates . 3 Both 
studies highlight other complex domains that are interested in improving their 
own professional performance. Both studies reveal the need to employ a vari- 
ety of research methods to deal with complicated issues, and they suggest that 
there are lessons to be learned from other domains. Perhaps the most telling 
connection is that, because lives are at stake, surgeons and astronauts experi- 
ence tremendous internal and external social pressure to avoid failure. The 
same often holds for intelligence analysts. 

In addition, surgery and astronautics are highly selective and private disci- 
plines. Although their work is not secret, both groups tend to be shielded from 
the outside world: surgeons for reasons of professional selection, training, and 
the fiscal realities of malpractice liability; astronauts because their community 



2 National Aeronautics and Space Administration, Astronaut Fact Book. 

3 NASA has launched missions with the shuttle fleet 113 times since 1981 and has experienced 
two catastrophic failures. It is probable that both of those were mechanical/engineering failures 
and not the result of astronaut error. Surgical reporting methods vary from hospital to hospital, 
and it is often difficult to determine the specific causes of morbidity and mortality. One longitudi- 
nal study of all surgical procedures in one medical center puts the surgical error rates at that center 
between 2.7 percent and 7.5 percent. See Hunter McGuire, Shelton Horsley, David Salter, et al., 
“Measuring and Managing Quality of Surgery: Statistical vs. Incidental Approaches.” 
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is so small and the selection and training processes are so demanding. 4 Intelli- 
gence analysts share many of these organizational and professional circum- 
stances. 

The Intelligence Community is relatively small, highly selective, and 
largely shielded from public view. For its practitioners, intelligence work is a 
cognitively-demanding and high-risk profession that can lead to public policy 
that strengthens the nation or puts it at greater risk. Because the consequences 
of failure are so great, intelligence professionals continually feel significant 
internal and external pressure to avoid it. One consequence of this pressure is 
that there has been a long-standing bureaucratic resistance to putting in place a 
systematic program for improving analytical performance. According to 
71 percent of the people 1 interviewed, however, that resistance has dimin- 
ished significantly since September 2001. 

It is not difficult to understand the historical resistance to implementing 
such a performance improvement program. Simply put, a program explicitly 
designed to improve human performance implies that human performance 
needs improving, an allegation that risks considerable political and institu- 
tional resistance. Not only does performance improvement imply that the sys- 
tem is not optimal, the necessary scrutiny of practice and performance would 
require examining sources and methods in detail throughout the Intelligence 
Community. Although this scrutiny would be wholly internal to the commu- 
nity, the concept runs counter to a culture of secrecy and compartmentaliza- 
tion. 

The conflict between secrecy, a necessary condition for intelligence, and 
openness, a necessary condition for performance improvement, was a recur- 
ring theme I observed during this research. Any organization that requires 
secrecy to perform its duties will struggle with and often reject openness, even 
at the expense of efficacy. Despite this, and to their credit, a number of small 
groups within the Intelligence Community have tasked themselves with creat- 
ing formal and informal ties with the nation’s academic, non-profit, and indus- 
trial communities. In addition, there has been an appreciable increase in the 
use of alternative analyses and open-source materials. 

These efforts alone may not be sufficient to alter the historical culture of 
secrecy, but they do reinforce the idea that the Intelligence Community itself 
has a responsibility to reconsider the relationship between secrecy, openness, 
and efficacy. This is especially true as it relates to the community’s perfor- 
mance and the occurrence of errors and failure. External oversight and public 
debate will not solve these issues; the desire to improve the Intelligence Com- 



4 There are currently 109 active US astronauts and 36 management astronauts. See National Aero- 
nautics and Space Administration- Johnson Space Center career astronaut biographies. 
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munity’s performance needs to come from within. Once the determination has 
been found and the necessary policy guidelines put in place, it is incumbent 
upon the Intelligence Community to find and utilize the internal and external 
resources necessary to create a performance improvement infrastructure. 



Scope 

This project was designed explicitly as an applied research program. In 
many respects, it resembles an assessment of organizational needs and a gap 
analysis, in that it was intended to identify and describe conditions and vari- 
ables that affect intelligence analysis and then to identify needs, specifica- 
tions, and requirements for the development of tools, techniques, and 
procedures to reduce analytic error. Based on these findings, I was to make 
recommendations to improve analytic performance. 

In previous human performance-related research conducted in the military, 
medical, and astronautic fields, I have found in place — especially in the mili- 
tary — a large social science literature, an elaborate training doctrine, and well- 
developed quantitative and qualitative research programs. In addition to 
research literature and programs, these three disciplines have substantial per- 
formance improvement programs. This was not the case with the Intelligence 
Community. 

This is not to say that an intelligence literature does not exist but rather that 
the literature that does exist has been focused to a greater extent on case stud- 
ies than on the actual process of intelligence analysis . 5 The vast majority of 
the available literature is about history, international relations, and political 
science. Texts that address analytic methodology do exist, and it is worth not- 
ing that there are quantitative studies, such as that by Robert Folker, that com- 
pare the effectiveness of different analytic methods for solving a given 
analytic problem. Folker’s study demonstrates that objective, quantitative, and 
controlled research to determine the effectiveness of analytic methods is pos- 
sible . 6 

The literature that deals with the process of intelligence analysis tends to be 
personal and idiosyncratic, reflecting an individualistic approach to problem 
solving. This is not surprising. The Intelligence Community is made up of a 
variety of disciplines, each with its own analytic methodology. The organiza- 
tional assumption has been that, in a multidisciplinary environment, intelli- 



5 There are exceptions. See the appendix. 

6 MSgt. Robert D. Folker, Intelligence Analysis in Theater Joint Intelligence Centers. Folker’s 
study contains a methodological flaw in that it does not describe one of the independent variables 
(intuitive method), leaving the dependent variable (test scores) in doubt. 
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gence analysts would use analytic methods and tools from their own domain 
in order to analyze and solve intelligence problems. When interdisciplinary 
problems have arisen, the organizational assumption has been that a variety of 
analytic methods would be employed, resulting in a “best fit” synthesis. 

This individualistic approach to analysis has resulted in a great variety of 
analytic methods — I identified at least 160 in my research for this paper — but 
it has not led to the development of a standardized analytic doctrine. That is, 
there is no body of research across the Intelligence Community asserting that 
method X is the most effective method for solving case one and that method Y 
is the most effective method for solving case two. 7 

The utility of a standardized analytic doctrine is that it enables an organiza- 
tion to determine performance requirements, a standard level of institutional 
expertise, and individual performance metrics for the evaluation and develop- 
ment of new analytic methodologies. 8 Ultimately, without such an analytic 
baseline, one cannot assess the effectiveness of any new or proposed analytic 
method, tool, technology, reorganization, or intervention. Without standard- 
ized analytic doctrine, analysts are left to the rather slow and tedious process 
of trial and error throughout their careers. 

Generally, in research literature, one finds a taxonomy, or matrix, of the 
variables that affect the object under study. Taxonomies help to standardize 
definitions and inform future research by establishing a research “road map.” 
They point out areas of interest and research priorities and help researchers 
place their own research programs in context. In my search of the intelligence 
literature, I found no taxonomy of the variables that affect intelligence analy- 
sis. 

Following the literature review, I undertook to develop working definitions 
and a taxonomy in order to systematize the research process. Readers will find 
the working definitions in the first chapter. The second chapter highlights the 
the broader findings and implications of this ethnographic study. Because the 
first two chapters contain many quotes from my interviews and workshops, 
they illustrate the tone and nature of the post- 9/11 environment in which I 
worked. 

The taxonomy that grew out of this work was first described in an article for 
the CS1 journal. Studies in Intelligence, and is presented here as Chapter 



7 There is no single Intelligence Community basic analytic training program. There is, however, 
community use of advanced analytic courses at both the CIA University and the Joint Military 
Intelligence College. The Generic Intelligence Training Initiative is a recent attempt to standard- 
ize certain law enforcement intelligence analysis training programs through a basic law enforce- 
ment analyst training curriculum. The program has been developed by the Training Advisory 
Council, under the Counterdrug Intelligence Coordinating Group and the Justice Training Center. 

8 See the appendix. 
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Three. In addition to the normal journal review process, I circulated a draft of 
the taxonomy among 55 academics and intelligence professionals and incor- 
porated their suggestions in a revised version that went to press. This is not to 
assert that the taxonomy is final; the utility of any taxonomy is that it can be 
revised and expanded as new research findings become available. The chapter 
by Dr. Judith Meister Johnston that follows offers an alternative model — more 
complex and possibly more accurate than the traditional intelligence cycle — 
for looking at the dynamics of the intelligence process, in effect the interrela- 
tionships of many elements of the taxonomy 

The following chapters, prepared by me and other able colleagues, were 
developed around other variables in the taxonomy and offer suggestions for 
improvement in those specific areas. One of them — Chapter Five, on integrat- 
ing methodologists and substantive experts in research teams — also appeared 
in Studies in Intelligence. Chapter Nine contains several broad recommenda- 
tions, including suggestions for further research. 

To the extent possible, I tried to avoid using professional jargon. Even so, 
the reader will still find a number of specific technical terms, and, in those 
cases, I have included their disciplinary definitions as footnotes. 



A Work in Progress 

In some respects, it may seem strange or unusual to have an anthropologist 
perform this type of work rather than an industrial/organizational psychologist 
or some other specialist in professional performance improvement or business 
processes. The common perception of cultural anthropology is one of field- 
work among indigenous peoples. Much has changed during the past 40 years, 
however. Today, there are many practitioners and professional associations 
devoted to the application of anthropology and its field methods to practical 
problem-solving in modern or postindustrial society. 9 

It is difficult for any modern anthropological study to escape the legacy of 
Margaret Mead. She looms as large over 20th century anthropology as does 
Sherman Kent over the intelligence profession. Although Franz Boas is argu- 
ably the father of American anthropology and was Margaret Mead’s mentor, 
hers is the name everyone recognizes and connects to ethnography. 10 Chances 
are, if one has read anthropological texts, one has read Mead. 



9 The Society for Applied Anthropology and the National Association for the Practice of Anthro- 

pology section of the American Anthropological Association are the two principal anthropologi- 
cal groups. Another group is the Inter-University Seminar on Armed Forces and Society, a 
professional organization representing 700 social science fellows, including practicing anthropol- 
ogists, applying their research methods to issues in the military. 
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INTRODUCTION 



I mention Mead not only because my work draws heavily on hers, but also 
because of her impact on the discipline and its direction. She moved from tra- 
ditional cultural anthropological fieldwork in the South Pacific to problem- 
oriented applied anthropology during World War II. She was the founder of 
the Institute for Intercultural Studies and a major contributor to the Cold War 
RAND series that attempted to describe the Soviet character. She also pio- 
neered many of the research methods that are used in applied anthropology 
today. I mention her work also as an illustrative point. After two years of field 
research in the South Pacific, she wrote at least five books and could possibly 
have written more. 

As I look over the stacks of documentation for this study, it occurs to me 
that, given the various constraints of the fellowship, there is more material 
here than I will be able to address in any one text. There are the notes from 
489 interviews, direct observations, participant observations, and focus 
groups; there are personal letters, e-mail exchanges, and archival material; and 
there are my own notes tracking the progress of the work. Moreover, the field- 
work continues. As I write this, I am scheduling more interviews, more obser- 
vations, and yet more fieldwork. 

This text, then, is more a progress report than a final report in any tradi- 
tional sense. It reflects findings and recommendations to date and is in no way 
comprehensive. Finally, based as it is on my own research interests and 
research opportunities, it is but one piece of a much larger puzzle. 



10 Boas (1858-1942) developed the linguistic and cultural components of ethnology. His most 
notable work was Race, Language, and Culture (1940). 
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Part I 

Research Findings 



i 





eth»nog*ra*phy\n [F ethnographie, fr. ethno- + -graphie -graphy] (1834) : 
the study and systematic recording of human cultures: also: a descriptive work 
produced from such research. ( Merriam Webster’s Collegiate Dictionary, 
Eleventh Edition) 
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Chapter One 

Definitions 



Because I conducted human performance-related fieldwork before I came 
to this project, I carried into it a certain amount of experiential bias, or “cogni- 
tive baggage.” The research findings from those other studies could bias my 
perspective and research approach within the Intelligence Community. For 
example, surgeons and astronauts do not need to deal with intentionally 
deceptive data. Patients are not trying to “hide” their illnesses from surgeons, 
and spacecraft are not thinking adversaries intent on denying astronauts criti- 
cal pieces of information. This one difference may mean that intelligence 
analysis is much more cognitively challenging than the other two cases and 
that the requisite psychomotor skills are significantly less important. In an 
effort to counteract the biases of experience, I will attempt to be explicit about 
my own definitions in this work. 



Working Definitions 

The three main definitions used in this work do not necessarily represent 
definitions derived from the whole of the intelligence literature. Although 
some of the definitions used in this work are based on the Q-sort survey of the 
intelligence literature described later, some are based on the 489 interviews, 
focus groups, and two years of direct and participant observations collected 
during this project. 
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